Data Flows vs Semantic Models

Regardless of which Model interface is used, the underlying engine produces 3 main constructs:

  • Master Flow: where advanced users can construct complex master flows, incorporating multiple data flows and models, scripts, APIs, and more.
  • Data Flow: a set of functions and tools for importing, cleaning, embellishing and preparing data for analysis.
  • Data Model: a tool for describing the structure of your data so it can be easily and properly queried and analyzed in the analytics tools like Discover - this is know as a "Semantic" model.

Of the 3, the master flow is only exposed in Model Pro sessions.

The Data flow is designed as an end-user's "ETL" tool set. ETL is the industry term for data preparation : Extract, Transform and Load. Sometimes ETL operations can be quite complex and detailed. The Pyramid data flow tool set is designed to make these capabilities easier to use and access.

The Semantic Data Model is the tool that will lead a user through the steps needed to describe the database structure that will be queried in Discover and elsewhere.

Materialization vs Virtualization

Pyramid's semantic models are virtual. That means they act as a light layer against any data source WITHOUT needing to ingest or duplicate the data into Pyramid's internal engines. As such Pyramid's approach to data modeling is "data virtualization" - allowing users to query data sets seamlessly from Pyramid without needing to understand how the underlying data set is constructed or where it resides (a.k.a. "in place analytics ").

However, there are times when data needs to be blended ("mashed-up"), fixed or embellished. This mostly requires a new version of the data to be materialized which is what is done in the Master and Data Flow tools. Pyramid allows users to WRITE data back to multiple data engines, including its own In-Memory database (IMDB).

Regardless of whether or not the data is materialized, the same virtual data modeling layer is applied against the source (including Pyramid's IMDB).